{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "HDDM tutorial: Posterior Predictives"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this tutorial you will learn how to run posterior predictive checks in HDDM.\n",
      "\n",
      "A posterior predictive check is a very useful tool when you want to evaluate if your model can reproduce key patterns in your data. Specifically, you can define a summary statistic that describes the pattern you are interested in (e.g. accuracy in your task) and then simulate new data from the posterior of your fitted model. You can the apply the the summary statistic to each of the data sets you simulated from the posterior and see if the model does a good job of reproducing this pattern by comparing the summary statistics from the simulations to the summary statistic caluclated over the model.\n",
      "\n",
      "What is critical is that you do not only get a single summary statistic from the simulations but a whole distribution which captures the uncertainty in our model estimate.\n",
      "\n",
      "Lets do a simple analysis using simulated data. First, import HDDM."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import hddm"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Simulate data from known parameters and two conditions (easy and hard)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data, params = hddm.generate.gen_rand_data(params={'easy': {'v': 1, 'a': 2, 't': .3},\n",
      "                                                   'hard': {'v': 1, 'a': 2, 't': .3}})"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First, lets estimate the same model that was used to generate the data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "m = hddm.HDDM(data, depends_on={'v': 'condition'})\n",
      "m.sample(1000, burn=20)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-------          19%                  ] 190 of 1000 complete in 0.5 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [--------------   38%                  ] 389 of 1000 complete in 1.0 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------60%--                ] 600 of 1000 complete in 1.5 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------81%-----------       ] 816 of 1000 complete in 2.0 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------100%-----------------] 1000 of 1000 complete in 2.4 sec"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 3,
       "text": [
        "<pymc.MCMC.MCMC at 0x7f4d6245cf50>"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next, we'll want to simulate data from the model. By default, `post_pred_gen()` will use 500 parameter values from the posterior (i.e. posterior samples) and simulate a different data set for each parameter value."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "m.nodes_db"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>knode_name</th>\n",
        "      <th>stochastic</th>\n",
        "      <th>observed</th>\n",
        "      <th>subj</th>\n",
        "      <th>node</th>\n",
        "      <th>tag</th>\n",
        "      <th>depends</th>\n",
        "      <th>hidden</th>\n",
        "      <th>rt</th>\n",
        "      <th>response</th>\n",
        "      <th>subj_idx</th>\n",
        "      <th>condition</th>\n",
        "      <th>mean</th>\n",
        "      <th>std</th>\n",
        "      <th>2.5q</th>\n",
        "      <th>25q</th>\n",
        "      <th>50q</th>\n",
        "      <th>75q</th>\n",
        "      <th>97.5q</th>\n",
        "      <th>mc err</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>a</th>\n",
        "      <td>    a</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "      <td>          a</td>\n",
        "      <td>      ()</td>\n",
        "      <td>          []</td>\n",
        "      <td> False</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td>  NaN</td>\n",
        "      <td>   1.78769</td>\n",
        "      <td>  0.1002926</td>\n",
        "      <td>  1.601673</td>\n",
        "      <td>  1.715472</td>\n",
        "      <td>  1.785334</td>\n",
        "      <td>  1.858317</td>\n",
        "      <td>  1.998753</td>\n",
        "      <td>  0.004570178</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>v(easy)</th>\n",
        "      <td>    v</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "      <td>    v(easy)</td>\n",
        "      <td> (easy,)</td>\n",
        "      <td> [condition]</td>\n",
        "      <td> False</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> easy</td>\n",
        "      <td>  1.008411</td>\n",
        "      <td>  0.1952847</td>\n",
        "      <td> 0.5986136</td>\n",
        "      <td> 0.8849071</td>\n",
        "      <td>  1.002399</td>\n",
        "      <td>  1.139823</td>\n",
        "      <td>  1.391829</td>\n",
        "      <td>  0.006383801</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>v(hard)</th>\n",
        "      <td>    v</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "      <td>    v(hard)</td>\n",
        "      <td> (hard,)</td>\n",
        "      <td> [condition]</td>\n",
        "      <td> False</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> hard</td>\n",
        "      <td> 0.9162392</td>\n",
        "      <td>  0.1670085</td>\n",
        "      <td> 0.6177067</td>\n",
        "      <td> 0.7979494</td>\n",
        "      <td> 0.9152084</td>\n",
        "      <td>  1.027397</td>\n",
        "      <td>  1.255026</td>\n",
        "      <td>  0.005925507</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>t</th>\n",
        "      <td>    t</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> False</td>\n",
        "      <td>          t</td>\n",
        "      <td>      ()</td>\n",
        "      <td>          []</td>\n",
        "      <td> False</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td>  NaN</td>\n",
        "      <td> 0.3491182</td>\n",
        "      <td> 0.02083996</td>\n",
        "      <td> 0.3014986</td>\n",
        "      <td> 0.3362049</td>\n",
        "      <td> 0.3517727</td>\n",
        "      <td> 0.3640453</td>\n",
        "      <td> 0.3835566</td>\n",
        "      <td> 0.0009445816</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>wfpt(easy)</th>\n",
        "      <td> wfpt</td>\n",
        "      <td> False</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> wfpt(easy)</td>\n",
        "      <td> (easy,)</td>\n",
        "      <td> [condition]</td>\n",
        "      <td> False</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> easy</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>        NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>          NaN</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>wfpt(hard)</th>\n",
        "      <td> wfpt</td>\n",
        "      <td> False</td>\n",
        "      <td>  True</td>\n",
        "      <td> False</td>\n",
        "      <td> wfpt(hard)</td>\n",
        "      <td> (hard,)</td>\n",
        "      <td> [condition]</td>\n",
        "      <td> False</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> NaN</td>\n",
        "      <td> hard</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>        NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>       NaN</td>\n",
        "      <td>          NaN</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "           knode_name stochastic observed   subj        node      tag  \\\n",
        "a                   a       True    False  False           a       ()   \n",
        "v(easy)             v       True    False  False     v(easy)  (easy,)   \n",
        "v(hard)             v       True    False  False     v(hard)  (hard,)   \n",
        "t                   t       True    False  False           t       ()   \n",
        "wfpt(easy)       wfpt      False     True  False  wfpt(easy)  (easy,)   \n",
        "wfpt(hard)       wfpt      False     True  False  wfpt(hard)  (hard,)   \n",
        "\n",
        "                depends hidden   rt response subj_idx condition       mean  \\\n",
        "a                    []  False  NaN      NaN      NaN       NaN    1.78769   \n",
        "v(easy)     [condition]  False  NaN      NaN      NaN      easy   1.008411   \n",
        "v(hard)     [condition]  False  NaN      NaN      NaN      hard  0.9162392   \n",
        "t                    []  False  NaN      NaN      NaN       NaN  0.3491182   \n",
        "wfpt(easy)  [condition]  False  NaN      NaN      NaN      easy        NaN   \n",
        "wfpt(hard)  [condition]  False  NaN      NaN      NaN      hard        NaN   \n",
        "\n",
        "                   std       2.5q        25q        50q        75q      97.5q  \\\n",
        "a            0.1002926   1.601673   1.715472   1.785334   1.858317   1.998753   \n",
        "v(easy)      0.1952847  0.5986136  0.8849071   1.002399   1.139823   1.391829   \n",
        "v(hard)      0.1670085  0.6177067  0.7979494  0.9152084   1.027397   1.255026   \n",
        "t           0.02083996  0.3014986  0.3362049  0.3517727  0.3640453  0.3835566   \n",
        "wfpt(easy)         NaN        NaN        NaN        NaN        NaN        NaN   \n",
        "wfpt(hard)         NaN        NaN        NaN        NaN        NaN        NaN   \n",
        "\n",
        "                  mc err  \n",
        "a            0.004570178  \n",
        "v(easy)      0.006383801  \n",
        "v(hard)      0.005925507  \n",
        "t           0.0009445816  \n",
        "wfpt(easy)           NaN  \n",
        "wfpt(hard)           NaN  "
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%pdb"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Automatic pdb calling has been turned ON\n"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "m.gen_stats()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>mean</th>\n",
        "      <th>std</th>\n",
        "      <th>2.5q</th>\n",
        "      <th>25q</th>\n",
        "      <th>50q</th>\n",
        "      <th>75q</th>\n",
        "      <th>97.5q</th>\n",
        "      <th>mc err</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>a</th>\n",
        "      <td>   1.78769</td>\n",
        "      <td>  0.1002926</td>\n",
        "      <td>  1.601673</td>\n",
        "      <td>  1.715472</td>\n",
        "      <td>  1.785334</td>\n",
        "      <td>  1.858317</td>\n",
        "      <td>  1.998753</td>\n",
        "      <td>  0.004570178</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>v(easy)</th>\n",
        "      <td>  1.008411</td>\n",
        "      <td>  0.1952847</td>\n",
        "      <td> 0.5986136</td>\n",
        "      <td> 0.8849071</td>\n",
        "      <td>  1.002399</td>\n",
        "      <td>  1.139823</td>\n",
        "      <td>  1.391829</td>\n",
        "      <td>  0.006383801</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>v(hard)</th>\n",
        "      <td> 0.9162392</td>\n",
        "      <td>  0.1670085</td>\n",
        "      <td> 0.6177067</td>\n",
        "      <td> 0.7979494</td>\n",
        "      <td> 0.9152084</td>\n",
        "      <td>  1.027397</td>\n",
        "      <td>  1.255026</td>\n",
        "      <td>  0.005925507</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>t</th>\n",
        "      <td> 0.3491182</td>\n",
        "      <td> 0.02083996</td>\n",
        "      <td> 0.3014986</td>\n",
        "      <td> 0.3362049</td>\n",
        "      <td> 0.3517727</td>\n",
        "      <td> 0.3640453</td>\n",
        "      <td> 0.3835566</td>\n",
        "      <td> 0.0009445816</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 14,
       "text": [
        "              mean         std       2.5q        25q        50q        75q  \\\n",
        "a          1.78769   0.1002926   1.601673   1.715472   1.785334   1.858317   \n",
        "v(easy)   1.008411   0.1952847  0.5986136  0.8849071   1.002399   1.139823   \n",
        "v(hard)  0.9162392   0.1670085  0.6177067  0.7979494  0.9152084   1.027397   \n",
        "t        0.3491182  0.02083996  0.3014986  0.3362049  0.3517727  0.3640453   \n",
        "\n",
        "             97.5q        mc err  \n",
        "a         1.998753   0.004570178  \n",
        "v(easy)   1.391829   0.006383801  \n",
        "v(hard)   1.255026   0.005925507  \n",
        "t        0.3835566  0.0009445816  "
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "m.nodes_db.loc[['v(easy)', 'v(hard)'], 'node']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "v(easy)    v(easy)\n",
        "v(hard)    v(hard)\n",
        "Name: node, dtype: object"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "hddm.analyze.plot_posterior_nodes?"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 21
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "hddm.analyze.plot_posterior_nodes(m.nodes_db.loc[['v(easy)', 'v(hard)'], 'node'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 20
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "hddm.utils.post_pred_stats?"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import kabuki"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "kabuki.analyze.post_pred_stats??"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "hddm.utils.post_pred_stats(data, ppc_data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>observed</th>\n",
        "      <th>mean</th>\n",
        "      <th>std</th>\n",
        "      <th>SEM</th>\n",
        "      <th>MSE</th>\n",
        "      <th>credible</th>\n",
        "      <th>quantile</th>\n",
        "      <th>mahalanobis</th>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>stat</th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>accuracy</th>\n",
        "      <td>      0.89</td>\n",
        "      <td>   0.88416</td>\n",
        "      <td> 0.05968831</td>\n",
        "      <td> 3.410564e-05</td>\n",
        "      <td>   0.0035968</td>\n",
        "      <td> True</td>\n",
        "      <td>     50.8</td>\n",
        "      <td> 0.09784166</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean_ub</th>\n",
        "      <td>   1.12827</td>\n",
        "      <td>  1.142066</td>\n",
        "      <td>  0.1159049</td>\n",
        "      <td> 0.0001903301</td>\n",
        "      <td>  0.01362428</td>\n",
        "      <td> True</td>\n",
        "      <td>       48</td>\n",
        "      <td>  0.1190288</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std_ub</th>\n",
        "      <td> 0.7302517</td>\n",
        "      <td> 0.6021974</td>\n",
        "      <td>  0.1314891</td>\n",
        "      <td>   0.01639789</td>\n",
        "      <td>  0.03368726</td>\n",
        "      <td> True</td>\n",
        "      <td>     83.9</td>\n",
        "      <td>  0.9738774</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10q_ub</th>\n",
        "      <td>     0.549</td>\n",
        "      <td> 0.5826357</td>\n",
        "      <td> 0.04893073</td>\n",
        "      <td>  0.001131359</td>\n",
        "      <td> 0.003525574</td>\n",
        "      <td> True</td>\n",
        "      <td>     23.8</td>\n",
        "      <td>  0.6874143</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>30q_ub</th>\n",
        "      <td>    0.7068</td>\n",
        "      <td> 0.7601027</td>\n",
        "      <td> 0.07249302</td>\n",
        "      <td>  0.002841174</td>\n",
        "      <td> 0.008096413</td>\n",
        "      <td> True</td>\n",
        "      <td>     24.9</td>\n",
        "      <td>    0.73528</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50q_ub</th>\n",
        "      <td>     0.882</td>\n",
        "      <td> 0.9683445</td>\n",
        "      <td>  0.1074765</td>\n",
        "      <td>  0.007455368</td>\n",
        "      <td>  0.01900657</td>\n",
        "      <td> True</td>\n",
        "      <td>     22.7</td>\n",
        "      <td>  0.8033799</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>70q_ub</th>\n",
        "      <td>    1.1356</td>\n",
        "      <td>   1.27357</td>\n",
        "      <td>   0.160154</td>\n",
        "      <td>   0.01903562</td>\n",
        "      <td>  0.04468494</td>\n",
        "      <td> True</td>\n",
        "      <td>     19.3</td>\n",
        "      <td>  0.8614808</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>90q_ub</th>\n",
        "      <td>    2.0804</td>\n",
        "      <td>     1.911</td>\n",
        "      <td>  0.3030602</td>\n",
        "      <td>   0.02869622</td>\n",
        "      <td>   0.1205417</td>\n",
        "      <td> True</td>\n",
        "      <td>     73.6</td>\n",
        "      <td>  0.5589634</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean_lb</th>\n",
        "      <td> -1.362455</td>\n",
        "      <td> -1.137568</td>\n",
        "      <td>  0.2939832</td>\n",
        "      <td>   0.05057389</td>\n",
        "      <td>       0.137</td>\n",
        "      <td> True</td>\n",
        "      <td> 18.79397</td>\n",
        "      <td>  0.7649633</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std_lb</th>\n",
        "      <td> 0.4672909</td>\n",
        "      <td> 0.4615062</td>\n",
        "      <td>  0.2638967</td>\n",
        "      <td> 3.346287e-05</td>\n",
        "      <td>  0.06967494</td>\n",
        "      <td> True</td>\n",
        "      <td> 54.57286</td>\n",
        "      <td> 0.02192036</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>10q_lb</th>\n",
        "      <td>      0.83</td>\n",
        "      <td>  0.711961</td>\n",
        "      <td>  0.2148102</td>\n",
        "      <td>   0.01393321</td>\n",
        "      <td>  0.06007661</td>\n",
        "      <td> True</td>\n",
        "      <td> 82.31156</td>\n",
        "      <td>  0.5495039</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>30q_lb</th>\n",
        "      <td>     1.016</td>\n",
        "      <td> 0.8582909</td>\n",
        "      <td>  0.2492302</td>\n",
        "      <td>   0.02487215</td>\n",
        "      <td>  0.08698786</td>\n",
        "      <td> True</td>\n",
        "      <td> 80.40201</td>\n",
        "      <td>  0.6327846</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50q_lb</th>\n",
        "      <td>     1.364</td>\n",
        "      <td>  1.030618</td>\n",
        "      <td>  0.3046174</td>\n",
        "      <td>    0.1111438</td>\n",
        "      <td>   0.2039355</td>\n",
        "      <td> True</td>\n",
        "      <td> 88.74372</td>\n",
        "      <td>    1.09443</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>70q_lb</th>\n",
        "      <td>     1.378</td>\n",
        "      <td>  1.268506</td>\n",
        "      <td>  0.3818061</td>\n",
        "      <td>   0.01198898</td>\n",
        "      <td>   0.1577649</td>\n",
        "      <td> True</td>\n",
        "      <td> 67.03518</td>\n",
        "      <td>  0.2867796</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>90q_lb</th>\n",
        "      <td>     1.962</td>\n",
        "      <td>  1.661069</td>\n",
        "      <td>  0.5549828</td>\n",
        "      <td>   0.09055966</td>\n",
        "      <td>   0.3985656</td>\n",
        "      <td> True</td>\n",
        "      <td> 75.17588</td>\n",
        "      <td>  0.5422354</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 30,
       "text": [
        "           observed       mean         std           SEM          MSE  \\\n",
        "stat                                                                    \n",
        "accuracy       0.89    0.88416  0.05968831  3.410564e-05    0.0035968   \n",
        "mean_ub     1.12827   1.142066   0.1159049  0.0001903301   0.01362428   \n",
        "std_ub    0.7302517  0.6021974   0.1314891    0.01639789   0.03368726   \n",
        "10q_ub        0.549  0.5826357  0.04893073   0.001131359  0.003525574   \n",
        "30q_ub       0.7068  0.7601027  0.07249302   0.002841174  0.008096413   \n",
        "50q_ub        0.882  0.9683445   0.1074765   0.007455368   0.01900657   \n",
        "70q_ub       1.1356    1.27357    0.160154    0.01903562   0.04468494   \n",
        "90q_ub       2.0804      1.911   0.3030602    0.02869622    0.1205417   \n",
        "mean_lb   -1.362455  -1.137568   0.2939832    0.05057389        0.137   \n",
        "std_lb    0.4672909  0.4615062   0.2638967  3.346287e-05   0.06967494   \n",
        "10q_lb         0.83   0.711961   0.2148102    0.01393321   0.06007661   \n",
        "30q_lb        1.016  0.8582909   0.2492302    0.02487215   0.08698786   \n",
        "50q_lb        1.364   1.030618   0.3046174     0.1111438    0.2039355   \n",
        "70q_lb        1.378   1.268506   0.3818061    0.01198898    0.1577649   \n",
        "90q_lb        1.962   1.661069   0.5549828    0.09055966    0.3985656   \n",
        "\n",
        "         credible  quantile mahalanobis  \n",
        "stat                                     \n",
        "accuracy     True      50.8  0.09784166  \n",
        "mean_ub      True        48   0.1190288  \n",
        "std_ub       True      83.9   0.9738774  \n",
        "10q_ub       True      23.8   0.6874143  \n",
        "30q_ub       True      24.9     0.73528  \n",
        "50q_ub       True      22.7   0.8033799  \n",
        "70q_ub       True      19.3   0.8614808  \n",
        "90q_ub       True      73.6   0.5589634  \n",
        "mean_lb      True  18.79397   0.7649633  \n",
        "std_lb       True  54.57286  0.02192036  \n",
        "10q_lb       True  82.31156   0.5495039  \n",
        "30q_lb       True  80.40201   0.6327846  \n",
        "50q_lb       True  88.74372     1.09443  \n",
        "70q_lb       True  67.03518   0.2867796  \n",
        "90q_lb       True  75.17588   0.5422354  "
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "data, params = hddm.generate.gen_rand_data(params={'easy': {'v': 1, 'a': 2, 't': .3},'hard': {'v': 1, 'a': 2, 't': .3}})\n",
      "m = hddm.HDDM(data, depends_on={'v': 'condition'})\n",
      "m.sample(1000, burn=20)\n",
      "ppc_data = hddm.utils.post_pred_gen(m)\n",
      "ppc_compare = hddm.utils.post_pred_stats(data, ppc_data, call_compare=False)\n",
      "print ppc_compare"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-------          19%                  ] 199 of 1000 complete in 0.5 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [---------------  41%                  ] 411 of 1000 complete in 1.0 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------61%---               ] 617 of 1000 complete in 1.5 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------83%-----------       ] 831 of 1000 complete in 2.0 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------100%-----------------] 1000 of 1000 complete in 2.4 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------100%-----------------] 2 of 2 complete in 0.0 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [------------------------------------200%------------------------------------] 4 of 2 complete in 30.5 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [---------------------------------------------250%----------------------------------------------] 5 of 2 complete in 60.9 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "           observed       mean         std           SEM          MSE  \\\n",
        "stat                                                                    \n",
        "accuracy       0.89    0.88416  0.05968831  3.410564e-05    0.0035968   \n",
        "mean_ub     1.12827   1.142066   0.1159049  0.0001903301   0.01362428   \n",
        "std_ub    0.7302517  0.6021974   0.1314891    0.01639789   0.03368726   \n",
        "10q_ub        0.549  0.5826357  0.04893073   0.001131359  0.003525574   \n",
        "30q_ub       0.7068  0.7601027  0.07249302   0.002841174  0.008096413   \n",
        "50q_ub        0.882  0.9683445   0.1074765   0.007455368   0.01900657   \n",
        "70q_ub       1.1356    1.27357    0.160154    0.01903562   0.04468494   \n",
        "90q_ub       2.0804      1.911   0.3030602    0.02869622    0.1205417   \n",
        "mean_lb   -1.362455  -1.137568   0.2939832    0.05057389        0.137   \n",
        "std_lb    0.4672909  0.4615062   0.2638967  3.346287e-05   0.06967494   \n",
        "10q_lb         0.83   0.711961   0.2148102    0.01393321   0.06007661   \n",
        "30q_lb        1.016  0.8582909   0.2492302    0.02487215   0.08698786   \n",
        "50q_lb        1.364   1.030618   0.3046174     0.1111438    0.2039355   \n",
        "70q_lb        1.378   1.268506   0.3818061    0.01198898    0.1577649   \n",
        "90q_lb        1.962   1.661069   0.5549828    0.09055966    0.3985656   \n",
        "\n",
        "         credible  quantile mahalanobis  \n",
        "stat                                     \n",
        "accuracy     True      50.8  0.09784166  \n",
        "mean_ub      True        48   0.1190288  \n",
        "std_ub       True      83.9   0.9738774  \n",
        "10q_ub       True      23.8   0.6874143  \n",
        "30q_ub       True      24.9     0.73528  \n",
        "50q_ub       True      22.7   0.8033799  \n",
        "70q_ub       True      19.3   0.8614808  \n",
        "90q_ub       True      73.6   0.5589634  \n",
        "mean_lb      True  18.79397   0.7649633  \n",
        "std_lb       True  54.57286  0.02192036  \n",
        "10q_lb       True  82.31156   0.5495039  \n",
        "30q_lb       True  80.40201   0.6327846  \n",
        "50q_lb       True  88.74372     1.09443  \n",
        "70q_lb       True  67.03518   0.2867796  \n",
        "90q_lb       True  75.17588   0.5422354  \n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "/home/wiecki/envs/hddm/local/lib/python2.7/site-packages/numpy/core/_methods.py:55: RuntimeWarning: Mean of empty slice.\n",
        "  warnings.warn(\"Mean of empty slice.\", RuntimeWarning)\n",
        "/home/wiecki/envs/hddm/local/lib/python2.7/site-packages/numpy/core/_methods.py:79: RuntimeWarning: Degrees of freedom <= 0 for slice\n",
        "  warnings.warn(\"Degrees of freedom <= 0 for slice\", RuntimeWarning)\n"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ppc_data = hddm.utils.post_pred_gen(m)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [-----------------100%-----------------] 2 of 2 complete in 0.0 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [------------------------------------200%------------------------------------] 4 of 2 complete in 24.1 sec"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r",
        " [---------------------------------------------250%----------------------------------------------] 5 of 2 complete in 47.9 sec"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The returned data structure is a pandas `DataFrame` object with a hierarchical index."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ppc_data.head(10)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th>rt</th>\n",
        "      <th>response</th>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>node</th>\n",
        "      <th>sample</th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "      <th></th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th rowspan=\"10\" valign=\"top\">wfpt(easy)</th>\n",
        "      <th rowspan=\"10\" valign=\"top\">0</th>\n",
        "      <th>0</th>\n",
        "      <td> 0.41009</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 0.79089</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td>-0.67769</td>\n",
        "      <td> 0</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 0.49359</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 1.59039</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>5</th>\n",
        "      <td> 0.99669</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>6</th>\n",
        "      <td> 5.51089</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>7</th>\n",
        "      <td> 0.73069</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>8</th>\n",
        "      <td> 0.82829</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>9</th>\n",
        "      <td> 0.92839</td>\n",
        "      <td> 1</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "                          rt  response\n",
        "node       sample                     \n",
        "wfpt(easy) 0      0  0.41009         1\n",
        "                  1  0.79089         1\n",
        "                  2 -0.67769         0\n",
        "                  3  0.49359         1\n",
        "                  4  1.59039         1\n",
        "                  5  0.99669         1\n",
        "                  6  5.51089         1\n",
        "                  7  0.73069         1\n",
        "                  8  0.82829         1\n",
        "                  9  0.92839         1"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The first level of the `DataFrame` contains each observed node. In this case the easy condition. If we had multiple subjects we would get one for each subject.\n",
      "\n",
      "The second level contains the simulated data sets. Since we simulated 500, these will go from 0 to 499 -- each with generated from a different parameter value sampled from the posterior.\n",
      "\n",
      "The third level is the same index as used in the data and numbers each trial in your data.\n",
      "\n",
      "For more information on how to work with hierarchical indices, see the [Pandas documentation](http://pandas.pydata.org/pandas-docs/stable/indexing.html#hierarchical-indexing-multiindex).\n",
      "\n",
      "There are also some helpful options like `append_data` you can pass to `post_pred_gen()`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "help(hddm.utils.post_pred_gen)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Help on function post_pred_gen in module kabuki.analyze:\n",
        "\n",
        "post_pred_gen(model, groupby=None, samples=500, append_data=False, progress_bar=True)\n",
        "    Run posterior predictive check on a model.\n",
        "    \n",
        "    :Arguments:\n",
        "        model : kabuki.Hierarchical\n",
        "            Kabuki model over which to compute the ppc on.\n",
        "    \n",
        "    :Optional:\n",
        "        samples : int\n",
        "            How many samples to generate for each node.\n",
        "        groupby : list\n",
        "            Alternative grouping of the data. If not supplied, uses splitting\n",
        "            of the model (as provided by depends_on).\n",
        "        append_data : bool (default=False)\n",
        "            Whether to append the observed data of each node to the replicatons.\n",
        "        progress_bar : bool (default=True)\n",
        "            Display progress bar\n",
        "    \n",
        "    :Returns:\n",
        "        Hierarchical pandas.DataFrame with multiple sampled RT data sets.\n",
        "        1st level: wfpt node\n",
        "        2nd level: posterior predictive sample\n",
        "        3rd level: original data index\n",
        "    \n",
        "    :See also:\n",
        "        post_pred_stats\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we want to compute the summary statistics over each simulated data set and compare that to the summary statistic of our actual data by calling `post_pred_stats()`."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ppc_compare = hddm.utils.post_pred_stats(data, ppc_data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print ppc_compare"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "          observed      mean       std       SEM       MSE  credible  \\\n",
        "stat                                                                   \n",
        "accuracy  0.890000  0.874580  0.063930  0.000238  0.004325         1   \n",
        "mean_ub   1.084831  1.048314  0.111169  0.001334  0.013692         1   \n",
        "std_ub    0.654891  0.542704  0.129186  0.012586  0.029275         1   \n",
        "10q_ub    0.510200  0.549030  0.045206  0.001508  0.003551         1   \n",
        "30q_ub    0.649200  0.704437  0.067714  0.003051  0.007636         1   \n",
        "50q_ub    0.818000  0.891622  0.099117  0.005420  0.015244         1   \n",
        "70q_ub    1.253800  1.165027  0.149496  0.007881  0.030230         1   \n",
        "90q_ub    1.884400  1.741424  0.282329  0.020442  0.100152         1   \n",
        "mean_lb  -0.970818 -1.046499  0.269939  0.005728  0.078595         1   \n",
        "std_lb    0.543502  0.423627  0.251797  0.014370  0.077772         1   \n",
        "10q_lb    0.547000  0.660271  0.203289  0.012830  0.054157         1   \n",
        "30q_lb    0.648000  0.785022  0.223779  0.018775  0.068852         1   \n",
        "50q_lb    0.693000  0.939898  0.269239  0.060958  0.133448         1   \n",
        "70q_lb    1.022000  1.158692  0.346341  0.018685  0.138637         1   \n",
        "90q_lb    1.666000  1.532752  0.515372  0.017755  0.283363         1   \n",
        "\n",
        "           quantile  mahalanobis  \n",
        "stat                              \n",
        "accuracy  55.500000     0.241202  \n",
        "mean_ub   64.699997     0.328490  \n",
        "std_ub    81.500000     0.868420  \n",
        "10q_ub    18.900000     0.858949  \n",
        "30q_ub    20.700001     0.815736  \n",
        "50q_ub    23.400000     0.742775  \n",
        "70q_ub    73.500000     0.593812  \n",
        "90q_ub    71.500000     0.506417  \n",
        "mean_lb   57.517658     0.280362  \n",
        "std_lb    72.754791     0.476077  \n",
        "10q_lb    26.538849     0.557192  \n",
        "30q_lb    25.933401     0.612307  \n",
        "50q_lb    14.228052     0.917022  \n",
        "70q_lb    40.060543     0.394675  \n",
        "90q_lb    65.893036     0.258547  \n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As you can see, we did not have to define the summary statistics as by default, `HDDM` already calculates a bunch of useful statistics for RT analysis such as the accuracy, mean RT of the upper and lower boundary (ub and lb respectively), standard deviation and quantiles. These are listed in the rows of the DataFrame.\n",
      "\n",
      "For each distribution of summary statistics there are multiple ways to compare them to the summary statistic obtained on the observerd data. These are listed in the columns. `observed` is just the value of the summary statistic of your data. `mean` is the mean of the summary statistics of the simulated data sets (they should be a good match if the model reproduces them). `std` is a measure of how much variation is produced in the summary statistic.\n",
      "\n",
      "The rest of the columns are measures of how far the summary statistic of the data is away from the summary statistics of the simulated data. `SEM` = standard error from the mean, `MSE` = mean-squared error, `credible` = in the 95% credible interval."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Finally, we can also tell `post_pred_stats()` to return the summary statistics themselves by setting `call_compare=False`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ppc_stats = hddm.utils.post_pred_stats(data, ppc_data, call_compare=False)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print ppc_stats.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "                 accuracy   mean_ub    std_ub    10q_ub    30q_ub    50q_ub  \\\n",
        "(wfpt(easy), 0)      0.96  1.164858  0.825420  0.500940  0.736800  0.909940   \n",
        "(wfpt(easy), 1)      0.92  1.066229  0.500696  0.552553  0.725853  0.842753   \n",
        "(wfpt(easy), 2)      0.84  1.106792  0.660981  0.538767  0.708527  0.852747   \n",
        "(wfpt(easy), 3)      0.90  0.949962  0.524693  0.507878  0.634398  0.784638   \n",
        "(wfpt(easy), 4)      0.88  0.967202  0.523246  0.509131  0.638661  0.781231   \n",
        "\n",
        "                   70q_ub    90q_ub   mean_lb    std_lb    10q_lb    30q_lb  \\\n",
        "(wfpt(easy), 0)  1.388660  1.902310 -1.270140  0.592450  0.796180  1.033160   \n",
        "(wfpt(easy), 1)  1.298903  1.815803 -0.921803  0.204067  0.720703  0.842803   \n",
        "(wfpt(easy), 2)  1.137547  1.791117 -1.610109  1.577114  0.813307  0.843867   \n",
        "(wfpt(easy), 3)  1.013418  1.533458 -1.125698  0.371009  0.667518  1.004518   \n",
        "(wfpt(easy), 4)  0.958081  1.826761 -0.765531  0.363230  0.545531  0.599181   \n",
        "\n",
        "                   50q_lb    70q_lb    90q_lb  \n",
        "(wfpt(easy), 0)  1.270140  1.507120  1.744100  \n",
        "(wfpt(easy), 1)  0.899403  0.964963  1.140823  \n",
        "(wfpt(easy), 2)  1.084597  1.183127  2.654677  \n",
        "(wfpt(easy), 3)  1.334438  1.347158  1.454438  \n",
        "(wfpt(easy), 4)  0.614681  0.665531  1.136381  \n"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This `DataFrame` has a row for each simulated data set. The columns are the different summary statistics."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Defining your own summary statistics"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can also define your own summary statistics and pass them to `post_pred_stats()`:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ppc_stats = hddm.utils.post_pred_stats(data, ppc_data, stats=lambda x: np.mean(x), call_compare=False)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ppc_stats.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>stat</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>(wfpt(easy), 0)</th>\n",
        "      <td> 1.067459</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>(wfpt(easy), 1)</th>\n",
        "      <td> 0.907187</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>(wfpt(easy), 2)</th>\n",
        "      <td> 0.672088</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>(wfpt(easy), 3)</th>\n",
        "      <td> 0.742396</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>(wfpt(easy), 4)</th>\n",
        "      <td> 0.759274</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 13,
       "text": [
        "                     stat\n",
        "(wfpt(easy), 0)  1.067459\n",
        "(wfpt(easy), 1)  0.907187\n",
        "(wfpt(easy), 2)  0.672088\n",
        "(wfpt(easy), 3)  0.742396\n",
        "(wfpt(easy), 4)  0.759274"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that `stats` can also be a dictionary mapping the name of the summary statistic to its function."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Summary statistics relating to outside variables"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Another useful way to apply posterior predictive checks is if you have trial-by-trial measure (e.g. EEG brain measure). In that case the `append_data` keyword argument is useful.\n",
      "\n",
      "Lets add a dummy column to our data. This is going to be uncorrelated to anything but you'll get the idea."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from numpy.random import randn\n",
      "data['trlbytrl'] = randn(len(data))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "m_reg = hddm.HDDMRegressor(data, 'v ~ trlbytrl')\n",
      "m_reg.sample(1000, burn=20)\n",
      "\n",
      "ppc_data = hddm.utils.post_pred_gen(m_reg, append_data=True)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " \r",
        "[                  0%                  ]  0 of 1 complete"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "linregress?"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 23
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from scipy.stats import linregress\n",
      "ppc_regression = []\n",
      "for (node, sample), sim_data in ppc_data.groupby(level=(0, 1)):\n",
      "    ppc_regression.append(linregress(sim_data.trlbytrl, sim_data.rt_sampled)[0]) # slope\n",
      "    \n",
      "orig_regression = linregress(data.trlbytrl, data.rt)[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 30
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.hist(ppc_regression)\n",
      "plt.axvline(orig_regression, c='r', lw=3)\n",
      "plt.xlabel('slope')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 35,
       "text": [
        "<matplotlib.text.Text at 0xc861f0c>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAEPCAYAAACneLThAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGVhJREFUeJzt3X1wVNX9x/HP0sQyLYQHHzYloU0bkoaFkESBDG3RjXRj\nwZLJoKaidjKAdjq001JbK7VOG+pPE6DOFOswzlS0W7Q8tH8kTNWMUVkUHQQFxZpSLCRDEpJ0IAQC\nCJFwfn+gW0JC2Ny9u0sO79dMZpK7d+/3e1z2k+PJ3Xs9xhgjAICVhiW6AQBA7BDyAGAxQh4ALEbI\nA4DFCHkAsBghDwAWGzDkFy5cKK/Xq9zc3D6PPf744xo2bJg6OjrC2yorK5WVlaWcnBy9/PLL7ncL\nABiUAUN+wYIFqq2t7bO9qalJdXV1+spXvhLeVl9frw0bNqi+vl61tbVavHixzp49637HAICIDRjy\nM2fO1JgxY/psv//++7VixYpe22pqajR//nwlJycrIyNDEyZM0Pbt293tFgAwKINek6+pqVF6erqm\nTJnSa/vBgweVnp4e/jk9PV0tLS3RdwgAcCxpMDufPHlSjz32mOrq6sLbBroqgsfjcd4ZACBqgwr5\nffv2qbGxUXl5eZKk5uZm3XDDDXr77beVlpampqam8L7Nzc1KS0vrc4wJEyZo3759UbYNAFeWzMxM\n/ec//xn8E80lNDQ0mMmTJ/f7WEZGhjl8+LAxxpgPP/zQ5OXlmdOnT5v9+/ebr33ta+bs2bN9nhNB\nySHtt7/9baJbiCnGdwnS/74uQza/fjaPzRjn2Tngmvz8+fP1jW98Q3v37tX48eP17LPP9nr8/OUY\nn8+nsrIy+Xw+zZ49W6tXr2a5BgASbMDlmnXr1g345P379/f6+aGHHtJDDz0UfVcAAFfwiVeX+f3+\nRLcQU4xvaLN5fDaPLRqeT9d64lfQ4xnwjBxgSDt/iZJ/53CR0+xkJg8AFiPkAcBihDwAWIyQBwCL\nEfIAYDFCHgAsRsgDgMUIeQCwGCEPABYj5AHAYoQ8AFiMkAcAixHyAGAxQh4ALEbIA4DFCHkAsBgh\nDwAWI+QBwGKEPABYjJAHAIsR8gBgsQFDfuHChfJ6vcrNzQ1ve+CBBzRx4kTl5eVp3rx5Onr0aPix\nyspKZWVlKScnRy+//HLsugaikJIyVh6PJyZf5+vv8ZSUsQkaNa5UA4b8ggULVFtb22tbcXGxPvzw\nQ73//vvKzs5WZWWlJKm+vl4bNmxQfX29amtrtXjxYp09ezZ2nQMOdXUdkWRi9HW+vo+fqw3Ez4Ah\nP3PmTI0ZM6bXtkAgoGHDzj2tsLBQzc3NkqSamhrNnz9fycnJysjI0IQJE7R9+/YYtQ0AiERUa/LP\nPPOM5syZI0k6ePCg0tPTw4+lp6erpaUluu4AAFFJcvrERx99VFdddZXuuuuui+5z4RrlZyoqKsLf\n+/1++f1+p20AgJVCoZBCoVDUx3EU8n/+85/14osv6tVXXw1vS0tLU1NTU/jn5uZmpaWl9fv880Me\nANDXhRPgZcuWOTrOoJdramtrtXLlStXU1Gj48OHh7SUlJVq/fr26u7vV0NCgjz76SNOnT3fUFADA\nHQPO5OfPn68tW7bo0KFDGj9+vJYtW6bKykp1d3crEAhIkmbMmKHVq1fL5/OprKxMPp9PSUlJWr16\n9UWXawAA8eExxlx43ldsC3o8inNJoJdzk4/Y/Bs0+t/ExtNvDf79wxmn2cknXgHAYoQ8AFiMkAcA\nixHyAGAxQh4ALEbIA4DFCHkAsBghDwAWI+QBwGKEPABYzPGlhoFopaSM5U5JQIxx7RokTCyvIXOJ\nyjGry7VrECtcuwYA0AchDwAWI+QBwGKEPABYjJAHAIsR8gBgMUIeACxGyAOAxQh5ALAYIQ8AFiPk\nAcBiA4b8woUL5fV6lZubG97W0dGhQCCg7OxsFRcXq7OzM/xYZWWlsrKylJOTo5dffjl2XQMAIjJg\nyC9YsEC1tbW9tlVVVSkQCGjv3r2aNWuWqqqqJEn19fXasGGD6uvrVVtbq8WLF+vs2bOx6xwAcEkD\nhvzMmTM1ZsyYXts2bdqk8vJySVJ5ebmqq6slSTU1NZo/f76Sk5OVkZGhCRMmaPv27TFqGwAQiUGv\nybe3t8vr9UqSvF6v2tvbJUkHDx5Uenp6eL/09HS1tLS41CYAwImobhri8Xg+vSb4xR/vT0VFRfh7\nv98vv98fTRsAYJ1QKKRQKBT1cQYd8l6vV21tbUpNTVVra6uuu+46SVJaWpqamprC+zU3NystLa3f\nY5wf8gCAvi6cAC9btszRcQa9XFNSUqJgMChJCgaDKi0tDW9fv369uru71dDQoI8++kjTp0931BQA\nwB0DzuTnz5+vLVu26NChQxo/frx+97vfaenSpSorK9OaNWuUkZGhjRs3SpJ8Pp/Kysrk8/mUlJSk\n1atXD7iUAwCIPe7xioThHq9A5LjHKwCgD0IeACxGyAOAxQh5ALAYIQ8AFiPkAcBihDwAWIyQBwCL\nEfIAYDFCHgAsRsgDgMUIeQCwGCEPABYj5AHAYoQ8AFiMkAcAixHyAGAxQh4ALEbIA4DFCHkAsBgh\nDwAWS0p0A8CVJUkejyfuVUeOHKNjxzriXheJ5zHGmLgW9HgU55K4TJ0Lu0T8W4hdXaP/Bbin3xqJ\nGzPvu6HNaXY6Xq6prKzUpEmTlJubq7vuukunT59WR0eHAoGAsrOzVVxcrM7OTqeHBwC4wFHINzY2\n6k9/+pN27typDz74QD09PVq/fr2qqqoUCAS0d+9ezZo1S1VVVW73CwAYBEchn5KSouTkZJ08eVJn\nzpzRyZMnNW7cOG3atEnl5eWSpPLyclVXV7vaLABgcByF/NixY/Xzn/9cX/7ylzVu3DiNHj1agUBA\n7e3t8nq9kiSv16v29nZXmwUADI6js2v27dunP/zhD2psbNSoUaN0xx136Lnnnuu1j8fjuehZBBUV\nFeHv/X6//H6/kzYAwFqhUEihUCjq4zg6u2bDhg2qq6vT008/LUlau3attm3bptdee02bN29Wamqq\nWltbVVRUpD179vQuyNk1+BRn18QT77uhLq5n1+Tk5Gjbtm36+OOPZYzRK6+8Ip/Pp7lz5yoYDEqS\ngsGgSktLnRweAOASx+fJr1ixQsFgUMOGDdP111+vp59+Wl1dXSorK9OBAweUkZGhjRs3avTo0b0L\nMpPHp5jJxxPvu6HOaXbyYSgkDCEfT7zvhjqn2cllDa5wKSlj1dV1JNFtAIgRZvJXuMTNpqVEzmqZ\nyWOoiftlDQAAlz9CHgAsRsgDgMUIeQCwGCEPABYj5AHAYoQ8AFiMkAcAixHyAGAxQh4ALEbIA4DF\nCHkAsBghDwAWI+QBwGKEPABYjJAHAIsR8gBgMUIeACxGyAOAxQh5ALAYIQ8AFnMc8p2dnbr99ts1\nceJE+Xw+vf322+ro6FAgEFB2draKi4vV2dnpZq8AgEFyHPI//elPNWfOHP3rX//S7t27lZOTo6qq\nKgUCAe3du1ezZs1SVVWVm70CAAbJY4wxg33S0aNHVVBQoP379/fanpOToy1btsjr9aqtrU1+v197\n9uzpXdDjkYOSiBGPxyMpUa9HomrHrq6R57wq/dVI3Jh53w1tTrPT0Uy+oaFB1157rRYsWKDrr79e\n9913n06cOKH29nZ5vV5JktfrVXt7u5PDAwBckuTkSWfOnNHOnTv15JNPatq0aVqyZEmfpRmPx/Pp\nLLGvioqK8Pd+v19+v99JGwBgrVAopFAoFPVxHC3XtLW1acaMGWpoaJAkbd26VZWVldq/f782b96s\n1NRUtba2qqioiOWayxzLNe5iuQaxEtflmtTUVI0fP1579+6VJL3yyiuaNGmS5s6dq2AwKEkKBoMq\nLS11cngAgEsczeQl6f3339e9996r7u5uZWZm6tlnn1VPT4/Kysp04MABZWRkaOPGjRo9enTvgszk\nLyvM5N3FTB6x4jQ7HYe8U4T85YWQdxchj1iJ63INAGBoIOQBwGKEPABYjJAHAIsR8gBgMUIeACxG\nyAOAxQh5ALAYIQ8AFiPkAcBihDwAWIyQBwCLEfIAYDFCHgAsRsgDgMUIeQCwGCEPABYj5AHAYoQ8\nAFiMkAcAixHyAGAxQh4ALEbIA4DFogr5np4eFRQUaO7cuZKkjo4OBQIBZWdnq7i4WJ2dna40CQBw\nJqqQX7VqlXw+nzwejySpqqpKgUBAe/fu1axZs1RVVeVKkwAAZxyHfHNzs1588UXde++9MsZIkjZt\n2qTy8nJJUnl5uaqrq93pEkCUkuTxeBLylZIyNtGDv6IlOX3iz372M61cuVLHjh0Lb2tvb5fX65Uk\neb1etbe3R98hABeckWQSUrmry5OQujjHUcj/4x//0HXXXaeCggKFQqF+9/nst3h/Kioqwt/7/X75\n/X4nbQCAtUKh0EXzdTA85rO1lkF46KGHtHbtWiUlJenUqVM6duyY5s2bpx07digUCik1NVWtra0q\nKirSnj17ehf0eOSgJGLk3C/iRL0eiaodu7pG/5vYePqtYd+YI6nNez56TrPT0Zr8Y489pqamJjU0\nNGj9+vW6+eabtXbtWpWUlCgYDEqSgsGgSktLnRweAOASV86T/2xZZunSpaqrq1N2drZee+01LV26\n1I3DAwAccrRcE1VBlmsuKyzXuIvlmv5r856PXlyXawAAQwMhDwAWI+QBwGKEPABYjJAHAIs5vqwB\n3JOSMlZdXUcS3QYAC3EK5WXgyjyNMZG1OYUy3rV5z0ePUygBAH0Q8gBgMUIeACxGyAOAxQh5ALAY\nIQ8AFiPkAcBihDwAWIyQBwCLEfIAYDFCHgAsRsgDgMUIeQCwGCEPABYj5AHAYoQ8AFjMUcg3NTWp\nqKhIkyZN0uTJk/XEE09Ikjo6OhQIBJSdna3i4mJ1dna62iwAYHAc3Rmqra1NbW1tys/P1/Hjx3XD\nDTeourpazz77rK655hr98pe/1PLly3XkyBFVVVX1LsidofrgzlD21OXOUP3X5j0fvbjeGSo1NVX5\n+fmSpBEjRmjixIlqaWnRpk2bVF5eLkkqLy9XdXW1k8MDAFwS9Zp8Y2Ojdu3apcLCQrW3t8vr9UqS\nvF6v2tvbo24QAOBcUjRPPn78uG677TatWrVKI0eO7PWYx+P5dBmir4qKivD3fr9ffr8/mjYAwDqh\nUEihUCjq4zhak5ekTz75RN/97nc1e/ZsLVmyRJKUk5OjUCik1NRUtba2qqioSHv27OldkDX5PliT\nt6cua/L91+Y9H724rskbY7Ro0SL5fL5wwEtSSUmJgsGgJCkYDKq0tNTJ4QEALnE0k9+6datuvPFG\nTZkyJbwkU1lZqenTp6usrEwHDhxQRkaGNm7cqNGjR/cuyEy+D2by9tRlJt9/bd7z0XOanY6Xa5wi\n5Psi5O2pS8j3X5v3fPTiulwDABgaCHkAsBghDwAWI+QBwGKEPABYjJAHAIsR8gBgMUIeACwW1QXK\nbJOSMlZdXUcS3QYAuIZPvJ4ncZ88vRI/dZrI2nziNb6SJZ1JSOWRI8fo2LGOhNR2m9PsZCYPIMbO\nKFG/YLq6+r/c+ZWENXkAsBghDwAWI+QBwGKEPABYjJAHAIsR8gBgMUIeACxGyAOAxQh5ALAYn3gF\nYLGkTy9XEl+X0+UULquQb2tr05o1axJybZthw/ifGsA+ibmkwuV0OQXXQ762tlZLlixRT0+P7r33\nXj344IMRP/fNN9/UI4+sVXf37W63dUlXXfV83GsCQKy5GvI9PT368Y9/rFdeeUVpaWmaNm2aSkpK\nNHHixIiP8fnPT9Lp0//nZlsRGT78bZ0+3ejCkUKS/C4c53IVEuMbykKyd3wh2Ts251xdo9i+fbsm\nTJigjIwMJScn684771RNTY2bJYaAUKIbiLFQohuIsVCiG4ixUKIbiKFQohu4LLka8i0tLRo/fnz4\n5/T0dLW0tLhZAgAwCK4u10T7V+xhw4apu/tNpaTMdamjyJ06tSvuNQEg1lwN+bS0NDU1NYV/bmpq\nUnp6eq99MjMzL/nL4NSpf7jZ1iC58VfxZQmq69RgazsZn1u141HX+fg8A/wUWe1Y+qyum6/fYGvH\nWn9jS8x/b7dP3czMzHTWh5u3/ztz5oy+/vWv69VXX9W4ceM0ffp0rVu3blB/eAUAuMfVmXxSUpKe\nfPJJ3XLLLerp6dGiRYsIeABIoLjfyBsAED8x/5hnR0eHAoGAsrOzVVxcrM7Ozj77nDp1SoWFhcrP\nz5fP59OvfvWrWLflmkjG19TUpKKiIk2aNEmTJ0/WE088kYBOnYlkfJK0cOFCeb1e5ebmxrlDZ2pr\na5WTk6OsrCwtX768331+8pOfKCsrS3l5edq1a+j8Yf5SY9uzZ49mzJih4cOH6/HHH09Ah9G51Pie\nf/555eXlacqUKfrmN7+p3bt3J6BL5y41vpqaGuXl5amgoEA33HCDXnvttYEPaGLsgQceMMuXLzfG\nGFNVVWUefPDBfvc7ceKEMcaYTz75xBQWFpo33ngj1q25IpLxtba2ml27dhljjOnq6jLZ2dmmvr4+\nrn06Fenr9/rrr5udO3eayZMnx7M9R86cOWMyMzNNQ0OD6e7uNnl5eX1ejxdeeMHMnj3bGGPMtm3b\nTGFhYSJaHbRIxvbf//7X7Nixw/z61782v//97xPUqTORjO+tt94ynZ2dxhhjXnrppSHz2hkT2fiO\nHz8e/n737t0mMzNzwGPGfCa/adMmlZeXS5LKy8tVXV3d735f+MIXJEnd3d3q6enR2LFjY92aKyIZ\nX2pqqvLz8yVJI0aM0MSJE3Xw4MG49ulUpK/fzJkzNWbMmHi25lgkH9o7f9yFhYXq7OxUe3t7Itod\nlEjGdu2112rq1KlKTk5OUJfORTK+GTNmaNSoUZLOvXbNzc2JaNWRSMb3xS9+Mfz98ePHdc011wx4\nzJiHfHt7u7xeryTJ6/Ve9I1y9uxZ5efny+v1qqioSD6fL9atuSLS8X2msbFRu3btUmFhYTzai9pg\nxzcURPKhvf72GQphYfsHEgc7vjVr1mjOnDnxaM0VkY6vurpaEydO1OzZsy+5/OvK2TWBQEBtbW19\ntj/66KO9fvZ4PBc9d3TYsGF67733dPToUd1yyy0KhULy+/1utBc1N8Ynnfute/vtt2vVqlUaMWKE\n63065db4hopIx2AuOCdhKIx9KPQYjcGMb/PmzXrmmWf05ptvxrAjd0U6vtLSUpWWluqNN97Q97//\nff373/++6L6uhHxdXd1FH/N6vWpra1NqaqpaW1t13XXXDXisUaNG6dZbb9U777xz2YS8G+P75JNP\ndNttt+mee+5RaWlprFp1xM3XbyiI5EN7F+7T3NystLS0uPXoVCRjG8oiHd/u3bt13333qba2dsgs\nI0qDf/1mzpypM2fO6PDhw7r66qv73SfmyzUlJSUKBoOSpGAw2G/AHTp0KHzWxscff6y6ujoVFBTE\nujVXRDI+Y4wWLVokn8+nJUuWxLvFqEQyvqFm6tSp+uijj9TY2Kju7m5t2LBBJSUlvfYpKSnRX/7y\nF0nStm3bNHr06PCy1eUskrF95sL/UxkKIhnfgQMHNG/ePD333HOaMGFCgjp1JpLx7du3L/za7dy5\nU5IuGvCSYn92zeHDh82sWbNMVlaWCQQC5siRI8YYY1paWsycOXOMMca8//77pqCgwOTl5Znc3Fyz\nYsWKWLflmkjG98YbbxiPx2Py8vJMfn6+yc/PNy+99FIi245YJOMzxpg777zTfOlLXzJXXXWVSU9P\nN88880yiWo7Iiy++aLKzs01mZqZ57LHHjDHGPPXUU+app54K7/OjH/3IZGZmmilTpph33303Ua0O\n2qXG1traatLT001KSooZPXq0GT9+vOnq6kpky4NyqfEtWrTIjB07NvxemzZtWiLbHbRLjW/58uVm\n0qRJJj8/33zrW98y27dvH/B4fBgKACzGPe8AwGKEPABYjJAHAIsR8gBgMUIeACxGyAOAxQh5XFH8\nfr/efffdRLcBxA0hjyuKLdffASJFyMNaJ06c0K233qr8/Hzl5uZq48aNvR5ft26dpkyZotzcXC1d\nujS8fcSIEbr//vs1efJkffvb39ahQ4cknfs4+ezZszV16lTdeOONA14UCrhcEPKwVm1trdLS0vTe\ne+/pgw8+0He+853wYwcPHtTSpUu1efNmvffee9qxY0f4ut0nT57UtGnT9M9//lM33XSTli1bJkn6\nwQ9+oD/+8Y965513tHLlSi1evDgh4wIGg5CHtaZMmaK6ujotXbpUW7duVUpKiqRzF+basWOH/H6/\nrr76an3uc5/T3Xffrddff13Suctef+9735Mk3XPPPdq6datOnDiht956S3fccYcKCgr0wx/+sN/L\nMwOXG1cuNQxcjrKysrRr1y698MILevjhh3XzzTeHH7twXd4Y0+9a/Wfbz549qzFjxgype70CEjN5\nWKy1tVXDhw/X3XffrV/84hfhgPZ4PJo+fbq2bNmiw4cPq6enR+vXr9dNN90k6dxdyv72t79Jkv76\n179q5syZGjlypL761a/q73//u6Rz4T/UbhCNKxMhD2t98MEHKiwsVEFBgR555BE9/PDD4cdSU1NV\nVVWloqIi5efna+rUqZo7d66kc/fQ3L59u3JzcxUKhfSb3/xGkvT8889rzZo1ys/P1+TJk7Vp06aE\njAsYDC41DFxg5MiR6urqSnQbgCuYyQMX4Dx62ISZPABYjJk8AFiMkAcAixHyAGAxQh4ALEbIA4DF\nCHkAsNj/A3N3mu3Cr2ykAAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0xc495bcc>"
       ]
      }
     ],
     "prompt_number": 35
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As you can see, the simulated data sets have on average no correlation to our trial-by-trial measure (just as in the data) but we also get a nice sense of the uncertainty in our estimation."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}